Image capturing apparatus

ABSTRACT

An image capturing apparatus comprises: a first image sensor having a plurality of pixels each counts a number of entering photons and outputs a count value as a first image signal; a second image sensor having a plurality of pixels each outputs an electric signal corresponding to a charge amount obtained by performing photoelectric conversion on entering light as a second image signal; and a generator that generates an image by selecting one of the first image signal and the second image signal.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of application Ser. No. 16/531,733,filed Aug. 5, 2019, the entire disclosure of which is herebyincorporated by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image capturing apparatus providedwith an imaging device employing an avalanche photodiode.

Description of the Related Art

Image sensors using a charge accumulation method is generally used inconventional digital cameras, video cameras, the like. The chargeaccumulation method is a method in which light incident on a photodiode(PD) in a fixed period is captured as an analog quantity called avoltage value.

With the charge accumulation method, when light is incident on a PD ofeach pixel, the PD generates and accumulates a charge substantiallylinearly with respect to an amount of incident light. The chargeaccumulated in the PD is transferred to a floating diffusion (FD) unit,converted into a voltage, and amplified by a source-follower (SF). Thevoltage output from each pixel is converted into a digital signal by anAD converter, and output to the exterior.

With the charge accumulation method, it is known that when, for example,the voltage in the FD is amplified by the SF, the S/N ratio drops due toRandom Telegraph Signal (RTS) noise produced at the boundary of the SFgate.

Meanwhile, recent years have seen investigations into photon countingtype image sensor that use an avalanche phenomenon occurring whenavalanche photodiodes (APDs) are operated in Geiger mode to measure thenumber of incoming photons themselves, and thus is capable of handlingentering light as digital value.

When an APD is operated in Geiger mode, an observable current isproduced by the avalanche phenomenon when a single photon enters theAPD, for example. By converting the current into a pulse signal andcounting the number of pulse signals, the number of incoming photons canbe measured directly. As such, an improvement in the S/N ratio can beanticipated, without producing RTS noise. Japanese Patent Laid-Open No.2014-81253 discloses a distance-measurement sensor constituted by theAPDs of a plurality of pixels as an example of a sensing deviceemploying APDs.

In Geiger mode, since an avalanche phenomenon can be caused even byincidence of a single photon, it is also called Single Photon AvalancheDiode (SPAD).

Since it is necessary to apply a high electric field higher than thebreakdown voltage in order to operate the APD in Geiger mode and a largecurrent flows due to the avalanche phenomenon when photons are incident,a large power consumption has been a problem.

In addition, when shooting a high-brightness subject, a plurality ofphotons are incident in a short time (dead time) period such that aphoton is incident during an avalanche phenomenon caused by anotherphoton is in progress, and there is a problem that photons cannot becounted appropriately.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the abovesituation, and improves image quality by taking advantage of a photoncounting type image sensor, such as SPAD, and a charge accumulation typeimage sensor, such as CMOS sensor.

According to the present invention, provided is an image capturingapparatus comprising: a first image sensor having a plurality of pixelseach counts a number of entering photons and outputs a count value as afirst image signal; a second image sensor having a plurality of pixelseach outputs an electric signal corresponding to a charge amountobtained by performing photoelectric conversion on entering light as asecond image signal; and a generator that generates an image byselecting one of the first image signal and the second image signal.

Further, according to the present invention, provided is an imagecapturing apparatus comprising: an image sensor in which a plurality offirst pixels and a plurality of second pixels are arranged alternately,whereby each of the first pixels counts a number of entering photons andoutputs a count value as a first image signal and each of the secondpixels outputs an electric signal corresponding to a charge amountobtained by performing photoelectric conversion on entering light as asecond image signal; a generator that generates an image by selectingone of the first image signal and the second image signal; and anobtaining unit that obtaining a luminance value for each pixel, whereinthe generator includes a dynamic range expander that expands a dynamicrange using the first image signal and the second image signal, and thedynamic range expander performs, for each pixel, processing of selectingthe first image signal in a case where the luminance value is less thana predetermined first threshold value, selecting and synthesizing thefirst and second image signals in a case where the luminance value isequal to or greater than the first threshold value and less than apredetermined second threshold value which is larger than the firstthreshold value, and selecting the second image signal and adjusting itssensitivity in a case where the luminance value is equal to or greaterthan the second threshold value.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention, andtogether with the description, serve to explain the principles of theinvention.

FIG. 1A is a diagram showing examples of an appearance of an imagecapturing apparatus, and FIGS. 1B and 1C are a perspective view and afront view showing examples of appearances of a camera module accordingto an embodiment the present invention;

FIG. 2 is a block diagram showing a configuration of the image capturingapparatus according to the embodiment;

FIGS. 3A and 3B are diagrams showing a configuration of a SPAD-typeimage sensor and a circuit configuration of a pixel according to theembodiment;

FIGS. 4A and 4B are diagrams for explaining a principle of operation ofthe SPAD-type image sensor according to the embodiment;

FIGS. 5A and 5B are diagrams showing the configuration of a CMOS imagesensor and a circuit configuration of a pixel according to anembodiment;

FIG. 6 is a flowchart for explaining an image shooting operationaccording to a first embodiment;

FIG. 7 is a view showing an example of an appearance of an imagecapturing apparatus according to a modification of the first embodiment;

FIG. 8 is a block diagram showing a configuration of the image capturingapparatus according to the modification of the first embodiment;

FIG. 9 is a flowchart for explaining an image shooting operationaccording to the modification of the first embodiment;

FIGS. 10A and 10B are diagrams for explaining count errors in aSPAD-type imaging device;

FIGS. 11A and 11B are diagrams showing an example of an image in which acount error has occurred in a SPAD type image sensor.

FIG. 12 is a flowchart for explaining an image shooting operationaccording to a second embodiment;

FIG. 13 is a diagram for explaining a dynamic range expansion processaccording to a third embodiment;

FIG. 14 is a flowchart for explaining an image shooting operationaccording to the third embodiment; and

FIG. 15 is a flowchart for explaining an image shooting operationaccording to a fourth embodiment.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described indetail in accordance with the accompanying drawings.

First Embodiment

A first embodiment of the present invention will be described withreference to FIGS. 1A to 6. The image capturing apparatus of the presentembodiment is applicable to an electronic still camera with a movingimage function, a video camera, and the like.

FIG. 1A is a view showing an example of an appearance of the imagecapturing apparatus according to an embodiment of the present invention.As shown in FIG. 1A, the image capturing apparatus 1 of the presentembodiment has two optical units, a first imaging system 11 and a secondimaging system 12.

FIG. 2 is a block diagram showing a configuration of the image capturingapparatus 1. The capturing apparatus 1 includes the first imaging system11, a first image signal processor 115, a firstcompression/decompression circuit 116, the second imaging system 12, asecond image signal processor 124, and a secondcompression/decompression circuit 125. Furthermore, a controller 13, anoperation unit 14, an image display 15, and an image recording unit 16are provided.

The first imaging system 11 includes a first optical lens barrel 101(first imaging optical system), a shutter mechanism 102, and a firstimage sensor 103. The first optical lens barrel 101 includes a lens forconverging light from a subject onto the first image sensor 103, and afirst optical mechanism section 111 provided with an optical mechanismfor performing focus adjustment, changing optical magnification, andadjusting an amount of light with respect to the converged light.

The first optical mechanism section 111 is driven based on a controlsignal from the controller 13. The shutter mechanism 102 is configuredbetween the first optical lens barrel 101 and the first image sensor103, and controls an exposure time for exposing the first image sensor103 with light passing through the first optical lens barrel 101according to the control signal from the controller 13.

The first image sensor 103 is a SPAD type imaging device comprising ofpixels using APD, and an image shooting operation is performed accordingto the control signal from the controller 13, and an image signal whichis a count value of the number of incident photons is output. Theconfiguration of the first image sensor 103 will be described later.

The first image signal processor 115 performs image processing, such ascolor correction processing, AE (Auto Exposure) processing, whitebalance processing, and optical shading correction processing on theimage signal from the first image sensor 103 under the control of thecontroller 13. After performing the image processing, the first imagesignal processor 115 outputs an image signal and a control signal to thecontroller 13. The image signal and control signal output from the firstimage signal processor 115 are recorded in a RAM (not shown) configuredin the controller 13.

The first compression/decompression circuit 116 operates under thecontrol of the controller 13 and compresses and encodes the imagesignals recorded in the RAM in the controller 13 from the first imagesignal processor 115 in a predetermined data format such as JPEG. Also,the first compression/decompression circuit 116 decompresses and decodesencoded data of the still image supplied from the controller 13 is.Furthermore, the first compression/decompression circuit 116 may beconfigured such that the compression encoding/decompression decoding ofa moving image may be executable using the MPEG (Moving Picture ExpertsGroup) method or the like.

The second imaging system 12 is configured of a second optical lensbarrel 121 (second imaging optical system) and a second image sensor123. The second optical lens barrel 121 has a configuration similar tothat of the first optical lens barrel 101, and a second opticalmechanism section 122 performs optical control in response to a controlsignal from the controller 13.

The second image sensor 123 is a CMOS-type image sensor, and reads outan image signal from CMOS pixels to be described later by an XY readoutmethod, and outputs the image signal according to a control signal fromthe controller 13.

The second image signal processor 124 performs image processing similarto the image processing performed on the first image signal processor115 on the image signal from the second image sensor 123, and outputsthe image signal and a control signal to the controller 13. The imagesignal and control signal output from the second image signal processor124 are recorded in the RAM (not shown) configured in the controller 13.

The second compression/decompression circuit 125 performs processingsimilar to the processing performed on the firstcompression/decompression circuit 116 on the image signal processed bythe second image signal processor 124.

In the present embodiment, the first image signal processor 115 and thesecond image signal processor 124, and the firstcompression/decompression circuit 116 and the secondcompression/decompression circuit 125 will be described separately.However, the present invention is not limited to this, and a singleimaging signal processor and a single compression/decompression circuitmay be used to process image signals obtained from the first imagingsystem 11 and the second imaging system 12.

The controller 13 is, for example, a microcontroller including a CPU, aROM, a RAM, and the like, and centrally controls the components of theimage capturing apparatus 1 by executing a program stored in the ROM andthe like.

The operation unit 14 includes, for example, various operation keys suchas a shutter release button, a lever, a dial, and the like, and outputsa control signal to the controller 13 according to an input operation bythe user. The image display 15 is composed of a display device such asan LCD and an interface circuit to this, generates an image signal to bedisplayed on the display device from the image signal supplied from thecontroller 13, and supplies this signal to the display device to displaythe image.

The image recording unit 16 receives an image data file encoded by thefirst compression/decompression circuit 116 or the secondcompression/decompression circuit 125 from the controller 13, andrecords the image data file in, for example, a portable semiconductormemory, an optical disk, an HDD, a magnetic tape or other storagemedium, and the like. Also, the image recording unit 16 reads out dataspecified based on the control signal from the controller 13 from thestorage medium and outputs the data to the controller 13.

FIGS. 1B and 1C are diagrams showing configuration examples of theappearance of a camera module provided with an imaging system that canbe used for the image capturing apparatus 1 of the present embodiment.FIG. 1B is a perspective view of the camera module 200, and FIG. 1C is afront view of the camera module 200.

The camera module in the present embodiment is a camera module in whichtwo image sensors are mounted by connecting two imaging systems. Amodule may be called by another name such as a package.

The camera module 200 is configured by fixing the first imaging system11 and the second imaging system 12 by a connecting member 400 having arectangular plate shape. In the first imaging system 11, the firstoptical lens barrel 101 (lens unit) including the first opticalmechanism section 111, the shutter mechanism 102, the first image sensor103, and so forth, are mounted. Similar to the first imaging system 11,the second imaging system 12 includes the second optical lens barrel 121(lens unit) including the second optical mechanism section 122, thesecond image sensor 123, and the like.

The connecting member 400 has a rectangular plate shape whose contour islarger than the size in the planar direction when the lens unit of thefirst imaging system 11 and the lens unit of the second imaging system12 are placed side by side. Further, in the connecting member 400, arectangular insertion hole into which the lens unit of the first imagingsystem 11 is inserted and a rectangular insertion hole into which thelens unit of the second imaging system 12 is inserted are formedsymmetrically. The lens unit of the first imaging system 11 and the lensunit of the second imaging system 12 are respectively inserted and fixedin the two rectangular insertion holes formed through the connectingmember 400.

Next, with reference to FIGS. 3A and 3B, the configuration of aSPAD-type image sensor used as the first image sensor 103 and thecircuit configuration of the pixel will be described. As shown in FIG.3A, the first image sensor 103 has a stacked structure including asensor substrate 201 and a circuit substrate 202. In addition, althoughthe image sensor shall have a stacked structure in this embodiment, thepresent invention is not limited to this, and a single layer structuremay be used as long as it has the same function.

The sensor substrate 201 is formed with a pixel array in which aplurality of pixels 203 are arranged in matrix, and a Bayer color filterarray of R(red), G (green) and B (blue), for example, is arranged on therespective pixels 203.

In the circuit substrate 202, a pixel control circuit 204, a signalprocessing circuit 205, and a substrate memory 206 are formed. The pixelcontrol circuit 204 is electrically connected to each pixel 203 in thesensor substrate 201 by a bump or the like, outputs a control signal fordriving the pixel 203, and receives a pulse waveform which is a bufferoutput from the pixel 203.

The pixel control circuit 204 is provided with a counter for determiningthe presence/absence of a photon by comparing a threshold value Vth setin advance and the output of each pixel 203 and counting the number ofincident photons by counting the number of pulse waveforms which changesover the threshold value.

Count values counted by the pixel control circuit 204 are output to theoutside of the first image sensor 103 by the signal processing circuit205. The substrate memory 206 is a volatile memory such as a DRAM, andis used to temporarily hold data when processing a signal from the pixelcontrol circuit 204 with the signal processing circuit 205.

Next, the configuration of the pixel 203 will be described. FIG. 3B isan equivalent circuit diagram of the pixel 203 formed in the sensorsubstrate 201. The pixel 203 includes a quenching resistor 301, anavalanche photodiode (APD) 302 (light receiving element), and a buffer303.

A reverse bias voltage with a potential HVDD is applied to the APD 302via the quenching resistor 301. The potential HVDD at this time is setsuch that the reverse bias voltage is equal to or higher than thebreakdown voltage in order to operate the APD 302 in Geiger mode. Theoutput of the buffer 303 is input to a counter 304 in the pixel controlcircuit 204.

Here, the operation of the pixel 203 at the time of photon incidencewill be briefly described using FIGS. 4A and 4B.

FIG. 4A shows the current-voltage characteristics of the APD 302. In thepresent embodiment, the potential HVDD for applying a reverse biasvoltage exceeding the breakdown voltage is applied to the cathode of theAPD 302 via the quenching resistor 301, and the APD 302 is in the Geigermode. Here, when a photon enters the APD 302, a large current(photocurrent) flows in the APD 302 due to avalanche multiplication(operation A).

At the same time as the current flows, the reverse bias voltage islowered by the quenching resistor 301, the reverse bias voltage appliedto the APD 302 becomes less than the breakdown voltage, and theavalanche multiplication stops (operation B). When the avalanchemultiplication is stopped, the cathode of the APD 302 is again chargedby the potential HVDD and returns to Geiger mode (operation C).

The voltage change at the buffer input due to operations A to C isshaped into a pulse signal by the buffer 303 and measured by the counter304. By repeating this, it is possible to count the number of photonsincident on the APD 302. In operation of the APD 302 for shootinghigh-brightness subjects and a moving picture, since the avalanchemultiplication is repeated, power consumption due to a large currentflow caused by the avalanche multiplication through the quenchingresistor is an issue.

FIG. 4B is a schematic diagram showing the relationship between thepulse waveform of the output voltage due to avalanche multiplicationoutput from the APD 302 at the time of photon incidence and adetermination threshold value Vth for determining the incidence of aphoton with the horizontal axis as time. FIG. 4B shows the case wherethe potential HVDD that can cause a reverse bias voltage sufficientlyexceeding the breakdown voltage to be supplied to the APD 302 issupplied. Under this situation, the avalanche multiplication occurs suchthat a pulse waveform that changes over the determination thresholdvalue Vth of the counter is generated for each of a photon A (time t1),a photon B (time t2), and a photon C (time t3) incident on the APD 302,and these pulses are separated with respect to time. Therefore, thenumber of photon events can be counted in the case of FIG. 4B.

Next, the configuration of the CMOS image sensor used as the secondimage sensor 123 and the circuit configuration of the pixel will bedescribed with reference to FIGS. 5A and 5B.

FIG. 5A is a block diagram showing the configuration of a CMOS imagesensor having column AD converters. As shown in FIG. 5A, the CMOS typeimage sensor as the second image sensor 123 has, on its light receivingsurface, a plurality of pixels 501 arranged in a matrix of M rows and Ncolumns for converting light into charges and accumulating them. Each ofthe plurality of pixels 501 is provided with a color filter of any of R(red), Gr, Gb (green) and B (blue) in a Bayer arrangement.

Further, column signal lines 502 for transmitting an image signalcorresponding to the charge amount accumulated in each pixel 501 areformed for each column, and a column circuit 503 is connected in seriesfor each column signal line 502.

Each column circuit 503 is comprised of an amplifier, a CorrelatedDouble Sampling (CDS) circuit, and an analog-to-digital (AD) converter(not shown). Each AD converter converts an image signal which is ananalog signal into a digital signal and outputs the digital signal, andthe digital signals are sequentially output through a horizontal signalline 505 by a column scanning circuit 504 and input to the second imagesignal processor 124.

Further, a row scanning circuit 506 receives a timing control signalfrom a timing control circuit 507 that operates according to a controlsignal from the controller 13 and scans transfer signal lines 508, resetsignal lines 509, and row selection signal lines 510. Then, the pixelsignals output from respective pixels 501 are read out row by row to thecolumn signal lines 502 of the respective columns.

Next, the configuration of the pixel 501 will be explained. FIG. 5B isan equivalent circuit diagram of the pixel 501. In the pixel 501, aphotodiode PD51, a transfer transistor M52, an amplification transistorM53, a selection transistor M54, and a reset transistor M55 areprovided. Here, each transistor is a switch element configured by ann-channel MOSFET.

The transfer signal line 508, the reset signal line 509, and the rowselection signal line 510 are connected to the gates of the transfertransistor M52, the reset transistor M55, and the selection transistorM54, respectively. These signal lines extend in the horizontal directionto simultaneously drive the pixels 501 included in the same row, wherebya rolling shutter of line sequential operation type or a global shutterof all row simultaneous operation type is realized. Furthermore, thecolumn signal line 502 is connected to the source of the selectiontransistor M54, and one end of the column signal line 502 is groundedvia a constant current source 56.

The photodiode PD51 performs photoelectric conversion and accumulatesthe generated charge. The P side is grounded, and the N side isconnected to the source of the transfer transistor M52. When thetransfer transistor M52 is turned on, the charge accumulated in thephotodiode PD51 is transferred to a floating diffusion portion (FD) 57.Since the FD 57 has a parasitic capacitance C58, the charge isaccumulated in this portion.

The drain of the amplification transistor M53 is connected to a powersupply voltage Vdd, and the gate is connected to the FD 57. Theamplification transistor M53 converts the voltage of the FD 57 into anelectrical signal.

The selection transistor M54 is for selecting the pixel from which thesignal is read out row by row, and its drain is connected to the sourceof the amplification transistor M53, and its source is connected to thecolumn signal line 502. When the selection transistor M54 is turned on,the amplification transistor M53 and the constant current source 56constitute a source follower, so that a voltage corresponding to thevoltage of the FD 57 is output to the column signal line 502.

The drain of the reset transistor M55 is connected to the power supplyvoltage Vdd, and the source is connected to the FD 57. The resettransistor M55 resets the photodiode PD51 to the power supply voltageVdd via the FD 57 and the transfer transistor M52.

Next, with reference to FIG. 6, the operation flow of a still imageshooting and a moving image shooting in the image capturing apparatus 1of the present embodiment will be described.

Compared to the pixels 501 of the CMOS type in the second image sensor123 shown in FIGS. 5A and 5B, the pixels 203 of the SPAD type in firstimage sensor 103 shown in FIGS. 3A and 3B are not provided with thereset transistor M55 and the amplification transistor M53.

Therefore, in the first image sensor 103, kTC noise and RTS noise causedby each transistor are not generated, and the S/N ratio is excellentcompared to the second image sensor 123 of the CMOS type. Accordingly,the first image sensor is advantageous to shoot a still image on whicheffects of random noise for the image quality is high.

On the other hand, in the case of shooting moving images and performingcontinuous shooting in which a large number of images are takenrepeatedly, the power consumption due to large current caused byavalanche multiplication is large, and it is conceivable that the numberof images that can be shot decreases when the image capturing apparatusis driven by a battery, and the operation time becomes short.

Therefore, in the first embodiment, in a case where still image shootingwith the large number of pixels is selected, the SPAD type first imagesensor 103 is used, and in a case where moving image shooting with highpower consumption per unit time is selected, the CMOS type second imagesensor 123 is used.

When power is turned on to start image shooting operation, thecontroller 13 first determines in step S101 whether a still image modeis selected or a moving image mode is selected by user operation to theoperation unit 14 or the like. If the still image mode is selected, theprocess proceeds to step S102, and if the moving image mode is selected,the process proceeds to step S103.

In step S102, since the still image mode is selected, the first imagingsystem 11 is operated for still image shooting, and a still imageshooting operation using the SPAD type first image sensor 103 isstarted.

On the other hand, in step S103, since the moving image mode isselected, the second imaging system 12 is operated for moving imageshooting, and a moving image shooting operation using the CMOS typesecond image sensor 123 is started.

In this way, by using the first imaging system 11 and the second imagingsystem 12 properly according to the still image mode and the movingimage mode, still image shooting and moving image shooting can beperformed while suppressing the total power consumption in the imagecapturing apparatus 1.

Next, in step S104, the controller 13 determines whether the recordingoperation in the selected mode is to be performed in response to useroperation to the operation unit 14 or the like.

In the case where recording is to be performed, the process proceeds tostep S105 where the image signal obtained by shooting in the modedetermined in step S101 is recorded by the image recording unit 16, andthe process proceeds to step S106. On the other hand, in a case whererecording is not performed, the process directly shifts to step S106.

In step S106, it is determined by the controller 13 whether or notshooting is to be ended by user operation to the operation unit 14 orthe like. If shooting is to be continued, the process returns to stepS101, and if the shooting is to be ended, the process ends.

According to the first embodiment as described above, in the imagecapturing apparatus including the SPAD-type image sensor and theCMOS-type image sensor, it is possible to achieve both high imagequality and power saving by performing shooting by switching between theimage sensors in accordance with the case of performing the still imageshooting and the case of performing the moving image shooting.

In the present embodiment, which of the first image sensor 103(SPAD-type image sensor) or the second image sensor 123 (CMOS-type imagesensor) is used is controlled by selecting the still image mode and themoving image mode, however, the present invention is not limitedthereto. For example, the switching between the first image sensor 103and the second image sensor 123 may be performed in accordance with thepower consumption and the upper limit of the temperature allowed for theimage capturing apparatus, and a shooting mode such as difference inmoving image resolution, single shooting and continuous shooting, and soforth. That is, if a shooting mode for performing a high resolutionmoving image shooting is set, it is controlled to use the first imagesensor 103 (SPAD type image sensor), and if a shooting mode forperforming a low resolution moving image shooting is set, it iscontrolled to use the second image sensor 123 (CMOS type image sensor).Alternatively, if a single shooting mode is set, it is controlled to usethe first image sensor 103 (SPAD type image sensor), and if a continuousshooting mode is set, it is controlled to use the second image sensor123 (CMOS type image sensor).

Further, switching may be performed according to the shutter speed atthe time of still image shooting and the frame rate at the time ofmoving image shooting. That is, if still image shooting is to beperformed at a shutter speed higher than a predetermined shutter speed,it is controlled to use the first image sensor 103 (SPAD-type imagesensor), and if still image shooting is to be performed at a shutterspeed is equal to or lower than the predetermined shutter speed, it iscontrolled to use the second image sensor 123 (CMOS type image sensor).Furthermore, if the frame rate at the time of moving image shooting islarger than a predetermined frame rate, it is controlled to use thefirst image sensor 103 (SPAD type image sensor), and if the frame rateat the time of moving image shooting is equal to or less than thepredetermined frame rate, it is controlled to use the second imagesensor 123 (CMOS type image sensor).

Modification of First Embodiment

FIG. 7 is a view showing an example of an appearance of an imagecapturing apparatus according to a modification of the first embodiment.The image capturing apparatus 1 shown in FIG. 1A has a configuration inwhich light from a subject is incident on two image sensors through twooptical systems. On the other hand, in an image capturing apparatus 7shown in FIG. 7, the light from the subject enters through one opticalsystem, is reflected or divided by a light guiding member, such as amirror, in the image capturing apparatus 7, and is incident on two imagesensors.

FIG. 8 is a block diagram showing the configuration of the imagecapturing apparatus 7 shown in FIG. 7. In FIG. 8, the same constituentsas those in FIG. 1A are referred to by the reference numerals, and adescription thereof will be omitted.

As shown in FIG. 8, the image capturing apparatus 7 in this modificationincludes an optical lens barrel 81, a shutter mechanism 102, a firstimage sensor 103, a second image sensor 123, a controller 13, anoperation unit 14, an image display 15, an image recording unit 16, anda mirror unit 82. In this modification, it will be described by assumingthat a function of performing various signal processing upon receivingimage signals from the first image sensor 103 and the second imagesensor 123 and a function of compressing and expanding imaging data areinherent to the controller 13.

The optical lens barrel 81 includes a lens for converging light from asubject onto the first image sensor 103 or the second image sensor 123configured in the image capturing apparatus 7, and an optical mechanismsection 811. The optical mechanism section 811 is driven based on acontrol signal from the controller 13, and performs focus adjustment,changes the optical magnification, adjusts an amount of the incidentlight, and so forth.

The mirror unit 82 includes a mirror 821 and a mirror driving unit 822and serves to guide light entering through the optical lens barrel 81 tothe first image sensor 103 or the second image sensor 123.

The mirror driving unit 822 drives the mirror 821 by an actuator or thelike in accordance with a control signal from the controller 13. Thatis, driving is performed so that the mirror 821 is at a first position(mirror down) on the optical axis of the lens shown in FIG. 8 or at asecond position (mirror up) retracted from the optical axis by flipingup the mirror 821.

In a case where the mirror 821 is at the first position, the light fromthe optical lens barrel 81 is reflected to be incident on the secondimage sensor 123. On the other hand, in a case where the mirror 821 isin the second position, the light from the optical lens barrel 81directly enters the first image sensor 103.

At this time, the configuration position of the mirror 821 is opticallyequivalent with respect to the imaging surfaces of the first imagesensor 103 and the second image sensor 123. In other words, the firstimage sensor 103 disposed on a first imaging surface and the secondimage sensor 123 disposed on a second imaging surface are arranged inimaging planes which are optically conjugate to each other with respectto the subject via the optical lens barrel 81.

FIG. 9 is a flowchart for explaining an operation of the image capturingapparatus 7 having the configuration shown in FIG. 8. Note that, in FIG.9, processes similarly to those in FIG. 6 are referred to by the samestep numbers, and the explanation of those processes will be omitted asappropriate.

When shooting is started, in step S101, if the still image mode isselected, the process proceeds to step S901, and if the moving imagemode is selected, the process proceeds to step S903.

If the still image mode is selected, the mirror 821 is driven to thesecond position (mirror-up position) in step S901 to direct the lightfrom the optical lens barrel 81 to the first image sensor 103.

After driving the mirror 821 to the second position in step S901, theprocess transitions to step S902 to start a still image shootingoperation using the SPAD type first image sensor 103, and the flowtransitions to step S104.

On the other hand, if the moving image mode is selected, the mirror 821is driven to the first position (mirror down position) in step S903, andthe light from the optical lens barrel 81 is guided to the second imagesensor 123.

After driving the mirror 821 to the first position in step S903, theprocess transitions to step S904 to start a moving image shootingoperation using the CMOS type second image sensor 123, and the flowtransitions to step S104.

The processes after step S104 are the same as the processes describedabove in FIG. 6, thus the description thereof is omitted.

According to the modification of the first embodiment as describedabove, in addition to the same effect as that of the first embodiment,the image capturing apparatus can be configured with a single opticalsystem.

Note that the mirror 821 does not have to be a total reflection mirror,and a half mirror, for example, may be used to divide the luminous fluxfrom the optical lens barrel 81 so that the divided luminous fluxes aresimultaneously incident on the first image sensor 103 and the secondimage sensor 123. In that case, the same operation described withreference to FIG. 6 in the first embodiment can be performed with aconfiguration in which the mirror operation is unnecessary.

Second Embodiment

Next, a second embodiment of the present invention will be described. Inthe second embodiment of the present invention, the image capturingapparatus 1 described with reference to FIGS. 1A to 5B in the firstembodiment, or the image capturing apparatus 7 described with referenceto FIGS. 7 and 8 in the modification of the first embodiment may beused, thus the description thereof is omitted here.

In a case of using the image capturing apparatus 7, the secondembodiment can be applied by using a half mirror instead of the totalreflection mirror. The processing in the second embodiment will bedescribed below with reference to FIGS. 10A to 12.

FIGS. 10A and 10B, and FIGS. 11A and 11B are diagrams for explaining thecase where a count error occurs in a SPAD-type image sensor.

FIG. 10A is a schematic diagram showing the relationship between thepulse waveform of the output voltage by avalanche multiplication whenphotons are incident and a determination threshold value Vth, where thehorizontal axis represents time. Unlike the waveform in the normaloperation described in FIG. 4B, after a voltage changes over thedetermination threshold value Vth due to a photon D (time t4), beforethe avalanche multiplication in operation B described with reference toFIG. 4A stops, a photon E (time t5) enters. At this time, since thephoton E enters at time t5 before the waveform due to avalanchemultiplication occurring at time t4 does not exceed the determinationthreshold value Vth, the count operation for the photon E cannot beperformed.

In addition, since the state at the time of incidence of a photon F(time t6) is the same as that at the time t4 to t5, the photon F is notcounted similarly. As described above, in a case where the luminance ofa subject is high, photons enter continuously before the waveformexceeds the determination threshold value Vth, so the count valuebecomes smaller than the number of photons actually incident, and counterror (count saturation) occurs.

During a period from time t6 to time t7, no photon enters, and thevoltage once exceeds the determination threshold value Vth, thus thepulse waveform of the voltage with respect to a photon G (time t7)incident thereafter is counted.

FIG. 10B is a view showing the relationship between illuminance andcount values in the SPAD-type image sensor. As the illuminanceincreases, the number of photons increases, so the count value countedby the SPAD-type image sensor also increases proportionately. However,when the illuminance is M or more, the state from time t4 to time t6 inFIG. 10A occurs, which causes a count error (count saturation). When theilluminance further increases, the number of photons that simultaneouslyenter increases further, and the count error state continues, and theactual count value (solid line) becomes inversely proportional to theideal count value (broken line) when the illuminance exceeds theilluminance N.

When the illuminance is equal to or greater than the illuminance M, if ascene including a high-brightness subject as shown in FIG. 11A is shot,the illuminance of the high-brightness subject causes an image in whicha darkening occurs as shown in the gradation portion in FIG. 11B.

Accordingly, in the second embodiment, in the image obtained by theSPAD-type first image sensor 103, an image portion corresponding to thehigh-luminance subject in which the count error has occurred isinterpolated by using an image obtained by the CMOS-type second imagesensor 123 in accordance with the luminance of the subject.

An image shooting operation in the second embodiment will be describedwith reference to the flowchart in FIG. 12.

When the image capturing apparatus is powered on and shooting isstarted, the controller 13 first determines in step S201 whethershooting and recording are to be performed according to an instructionof shooting and recording by a user operation to the operation unit 14.

If it is determined in step S201 that shooting and recording are to beperformed, shooting is performed using the first image sensor 103 instep S203, and shooting is performed using the second image sensor 123in parallel, and images are acquired respectively. On the other hand, ifshooting and recording are not to be performed, the process proceeds tostep S209.

In the present embodiment, it is described that the first image sensor103 and the second image sensor 123 perform image shooting in parallel,but images may be acquired separately in time sequence.

Next, in step S204, based on an area distribution of image signal valuesin the image acquired from the second image sensor 123 in step S203, anaperture setting value of the second optical mechanism section 122, andan exposure period and a sensitivity setting value of the second imagesensor 123, luminance values of subjects are calculated by the secondimage signal processor 124 for each of the areas determined in advance.

Then, the luminance values of subjects are held in the controller 13 ascorrection determination values. The above-mentioned area may be setappropriately, for example, by detecting a subject/subjects using aknown method and setting each subject as each area, or dividing an imageinto a plurality of blocks of a predetermined size.

Next, in step S205, the controller 13 determines whether at least one ofthe correction determination values of the respective areas obtained instep S204 is equal to or greater than a correction determinationthreshold value at which a count error occurs in the first image sensor103. Then, if at least one of the correction determination values isequal to or greater than the correction determination threshold value,it is determined that the correction is necessary, and if all thecorrection determination values are less than the correctiondetermination threshold value, it is determined that the correction isunnecessary. The correction determination threshold value is held inadvance as a design value in the controller 13 based on, for example,the characteristics of the first image sensor 103 described withreference to FIG. 10B.

If it is determined in step S205 that the correction is unnecessary, theprocess proceeds to S206, and if it is determined that the correction isnecessary, the process proceeds to step S207.

In step S206, since the correction is not necessary, the image obtainedby the first image sensor 103 in step S202 is recorded as is by theimage recording unit 16, and the process proceeds to step S209.

On the other hand, in step S207, it is determined in step S205 that theimage obtained by the first image sensor 103 includes a pixel signalhaving a count error. Therefore, in the image obtained by the firstimage sensor 103, the controller 13 corrects image signals in addressesin the area whose correction determination value obtained in step S204exceeds the correction determination threshold value by exchanging theimage signals with the image signals in the same addresses obtained fromthe second image sensor 123. After the correction, the process proceedsto step S209.

The difference in signal level between the images obtained from thefirst image sensor 103 and the second image sensor 123 is, for example,obtained as a table or a function of the difference between the signallevels of the second image sensor 123 and the first image sensor 103obtained for each luminance, and convert it. In addition, if the area inwhich the count error occurs in the image obtained by the first imagesensor 103 is small, correction may be performed by interpolating thesignal values in the pixel area in which the count error occurs usingsurrounding pixel signals in which the count error does not occur, or byreplacing the signal values in the pixel area in which the count erroroccurs with a fixed value. Other various correction methods can beconsidered, and the present invention is not limited by the correctionmethod.

In step S209, the controller 13 determines whether or not to end theshooting, and if yes, the process transitions to the standby state, andif not, the process returns to step S201 and the operation is continued.

In the above example, the subject luminance value is calculated for eachregion and used as the correction determination value which is comparedwith the correction determination threshold value, however, the presentinvention is not limited to this. For example, the signal valuedifference or the signal value ratio of the same address area betweenthe images obtained by the first image sensor 103 and the second imagesensor 123 may be used as the correction determination values.Alternatively, the determination may be made by comparing the maximumvalue, as a correction determination value, of the subject luminancevalues acquired for each area with the correction determinationthreshold value.

Also, the maximum luminance value in the image obtained from the secondimage sensor 123 may be compared as correction determination values withthe correction determination threshold value, and if it is determinedthat correction is necessary, determination on the subject luminancevalue for each area may be performed. That is, the luminance value ofthe image obtained from the second image sensor 123 is used to determinewhether there is a count error in the image obtained from the firstimage sensor 103 and the correction is performed as needed. Variousother methods and procedures are conceivable.

According to the second embodiment as described above, it is possible tocorrect an image by detecting a count error caused by a high-brightnesssubject occurring in a case of using a SPAD-type image sensor, and thusit is possible to provide an image with good image quality.

Third Embodiment

Next, a third embodiment of the present invention will be described. Inthe third embodiment of the present invention, the image capturingapparatus 1 described with reference to FIGS. 1A to 5B in the firstembodiment, or the image capturing apparatus 7 described with referenceto FIGS. 7 and 8 in the modification of the first embodiment may beused, thus the description thereof is omitted here.

In a case of using the image capturing apparatus 7, the third embodimentcan be applied by using a half mirror instead of the total reflectionmirror. The processing in the third embodiment will be described belowwith reference to FIGS. 13 and 14.

In the third embodiment, an operation in which dynamic range expansionprocessing (hereinafter referred to as “HDR processing”) is performedusing image signals obtained from the SPAD type first image sensor 103and the CMOS type second image sensor 123 will be described. In thepresent embodiment, it is assumed that the HDR processing is performedin the controller 13.

FIG. 13 is a diagram showing the relationship between the amount ofincident light and the amount of signal in the HDR processing. Ingeneral, the HDR processing is processing for expanding the dynamicrange of a signal using images acquired under image shooting conditionsof low exposure and high exposure.

As described in the background of the invention and the description ofthe second embodiment, the SPAD-type imaging device has an advantage ofhigh S/N ratio because the RTD noise does not occur, for example,however, a count error tends to occur on the high luminance side.

Therefore, in the HDR processing in the third embodiment, the dynamicrange is expanded using an image signal from the CMOS type image sensorin the signal range where the amount of incident light is large, andusing an image signal from the SPAD type image sensor capable ofobtaining an image signal with a high S/N ratio in the low luminanceside where the amount of incident light is small.

In the SPAD type first image sensor 103, when the incident light amountreaches P2, the signal amount Q3 where the count error starts to occuris reached, and in the CMOS type second image sensor 123, the saturationsignal amount Q2 of the CMOS type image sensor is reached when theincident light amount becomes P3. On the other hand, in a case where thesignal amount obtained by light reception is Q1 or less, the pixelsignal cannot be used because it corresponds to the noise level.

Therefore, the dynamic range of the first image sensor 103 is in therange from P0 to P2 of the amount of incident light, and the dynamicrange of the second image sensor 123 is in the range from P1 to P3 ofthe amount of incident light.

Here, it is assumed that the ratio of the signal amounts of the firstimage sensor 103 and the second image sensor 123 is 3:1. In this case,the controller 13 obtains the pixel signal HDL_A after the HDRprocessing by the following equation (1) for the signal amount in therange of the incident light amount A (light amounts P0 to P1, low level)for a pixel in the imaging screen.

pixel signal HDL_A=pixel signal of first image sensor×1+pixel signal ofsecond image sensor×0   (1)

Further, the controller 13 obtains the pixel signal HDL_B after the HDRprocessing by the following equation (2) for the signal amount in therange of the incident light amount B (light amount P1 to P2, middlelevel) for a certain pixel in the imaging screen.

pixel signal HDL_B=pixel signal of first image sensor×(1-α)+pixel signalof second image sensor×α×3   (2)

Furthermore, the controller 13 obtains a pixel signal HDL_C after theHDR processing according to the following equation (3) for the signalamount in the range of the incident light amount C (light amount P2 toP3, high level) for a certain pixel in the imaging screen.

pixel signal HDL_C=pixel signal of first image sensor×0+pixel signal ofsecond image sensor×3   (3)

As described above, the controller 13 classifies the signal amount ofeach pixel in the imaging screen into, for example, three levels of lowlevel, middle level, and high level. Then, for the pixel signalcorresponding to the incident light amount that cause a low level ofsignal amount, the pixel signal after the HDR processing is obtained bythe equation (1) which uses only the pixel signal of the first imagesensor 103.

In addition, for a pixel signal corresponding to an incident lightamount that causes a medium level of signal amount, the controller 13combines the pixel signal of the first image sensor 103 and the pixelsignal of the second image sensor 123 at a ratio of (1-α):α using theequation (2) to obtain the pixel signal after the HDR processing isobtained. Here, α (α is 0 or more and 1 or less) represents a synthesisratio. Furthermore, for a pixel signal corresponds to an incident lightamount that causes a high level of signal amount, the controller 13obtains the pixel signal after the HDR processing using the equation (3)which uses only the pixel signal of the second image sensor 123.

As a result, as shown in FIG. 13, it is possible to generate a highdynamic range image in which the signal amount is expanded from Q1 toQ4. The low level, medium level, and high level sections of the signalamount level are determined in advance according to the characteristicsof the first image sensor 103 and the second image sensor 123.

Next, the HDR processing in the imaging device of the third embodimentwill be described with reference to the flowchart in FIG. 14.

When the shooting is started after the HDR mode is selected by the useroperation, predetermined shooting conditions are set for the first imagesensor 103 and the second image sensor 123 in step S301.

Next, in step S303, shooting is performed using the first image sensor103, and shooting is performed in parallel using the second image sensor123, and images are acquired respectively. In the present embodiment, itis described that the first image sensor 103 and the second image sensor123 perform image shooting in parallel, but images may be acquiredseparately in time sequence.

Next, in step S304, the controller 13 performs the above-described HDRprocessing using the images from the first image sensor 103 and thesecond image sensor 123 obtained in step S303.

Next, in step S305, it is determined whether or not the HDR processingis to be ended, and if it is to be ended, the transition to the standbystate is made. On the other hand, when the shooting is to be continued,the process returns to step S301 to continue the shooting.

In the present embodiment, although an imaging apparatus having atwo-image-sensor configuration in which a SPAD-type image sensor and aCMOS-type image sensor are separately configured is described, thepresent invention is not limited to this. For example, SPAD-type imagingpixels and CMOS-type imaging pixels are alternately arranged as asingle-image-sensor configuration, and HDR processing may be performedusing image signals obtained from rows of SPAD-type imaging pixels androws of CMOS-type imaging pixels, which are physically adjacent to eachother. At this time, the SPAD type imaging pixels and the CMOS typeimaging pixels may be alternately formed row by row or for everymultiple rows, or may be alternately formed column by column or forevery multiple columns, or may be formed in a checkered pattern.

In the example described above, although the pixel signal after the HDRprocessing is acquired using one of the formulas (1) to (3) according tothe incident light amount, the present invention is not limited to this,and which one of the equations (1) to (3) is to be used may bedetermined according to the signal amount. In that case, for example,the low level, the middle level, or the high level is determined basedon the signal amount obtained from the first image sensor 103 and thesignal amount obtained from the second image sensor 123, and one offormulas (1) to (3) is used based on the determination result.

According to the third embodiment as described above, by performing theHDR processing as described above using the SPAD-type image sensor andthe CMOS-type image sensor, it is possible to obtain an image of adynamic range suitable for the luminance of the subject.

Fourth Embodiment

Next, a fourth embodiment of the present invention will be described. Inthe fourth embodiment of the present invention, the image capturingapparatus 1 described with reference to FIGS. 1A to 5B in the firstembodiment, or the image capturing apparatus 7 described with referenceto FIGS. 7 and 8 in the modification of the first embodiment may beused, thus the description thereof is omitted here. The processing inthe fourth embodiment will be described below with reference to FIG. 15.

In the fourth embodiment, image shooting is performed by switchingbetween a SPAD-type image sensor and a CMOS-type image sensor accordingto a sensitivity setting in shooting conditions. The configuration ofthe pixel 203 of the SPAD type first image sensor 103 described withreference to FIGS. 3A and 3B does not need the reset transistor M55 andthe amplification transistor M53 in the configuration of the pixel 501of the CMOS second image sensor 123 described with reference to FIGS. 5Aand 5B.

Therefore, kTC noise and RTS noise due to these configurations do notoccur, so the S/N ratio of the SPAD type image sensor is superior tothat of the CMOS type image sensor. However, as described with referenceto FIGS. 10A and 10B in the second embodiment, a count error occursunder the condition that many photons are incident within apredetermined period of time.

In consideration of above, in the image capturing apparatus according tothe fourth embodiment, image shooting is performed using the CMOS imagesensor under the exposure condition in which a large amount of photonsare incident on pixels, and image shooting is performed using theSPAD-type image sensor under the exposure condition for low illuminationsince a large amount of noise tends to be generated.

First, when the imaging apparatus is powered on and image shooting isstarted, in step S401, an imaging operation by the second image sensor123 is performed in order to determine an exposure value. In step S401,since the condition regarding the brightness of the subject is not knownimmediately after the start of shooting, the image is acquired by theCMOS type second image sensor 123 which does not cause a problem withhigh brightness subject.

Next, in step S402, a subject luminance value Ex is obtained from theimage obtained in step S401. In the present embodiment, the subjectluminance value Ex is calculated based on the image signal acquired bythe second image sensor 123, the exposure time and sensitivity settingvalue set in the second image sensor 123, and the optical aperturesetting value in the second optical mechanism section 122.

In step S402, in a case where the calculation is performed based on theimage signal from the second image sensor 123, the calculation is doneby the second image signal processor 124. However, in a case where theprocess moves from step S408 which will be described later to step S402and the calculation is to be performed based on the image signal fromthe first image sensor 103, the calculation is performed by first imagesignal processor 115.

Next, in step S403, the controller 13 determines whether the subjectluminance value Ex calculated in step S402 is equal to or greater than apredetermined threshold value Eth. Here, as the threshold value Eth, thebrightness at which the SPAD type first image sensor 103 causes a counterror, or the brightness at which the S/N ratio of the CMOS type secondimage sensor 123 exceeds the allowable value is set.

If the subject luminance value Ex is less than the threshold value Ethin step S403, the process proceeds to step S404, and shooting isperformed using the SPAD type first image sensor 103. On the other hand,if the subject luminance value Ex is equal to or greater than thethreshold value Eth in step S403, the controller 13 transitions to stepS405, and shooting is performed using the CMOS type second image sensor123.

In step S406, the controller 13 determines whether the recordingoperation is to be performed based on a user operation to the operationunit 14 or the like. If recording is to be performed, the processproceeds to S407, the image signal obtained by the shooting in step S404or S405 is recorded by the image recording unit 16, and the processproceeds to step S408. On the other hand, if recording is not to beperformed, the process directly proceeds to step S408.

In step S408, the controller 13 determines whether or not to end theimage shooting based on the user operation to the operation unit 14 orthe like, and the controller 13 returns the process to step S402 ifcontinuing the shooting, and ends the process if ending the shooting.

When transitioning the process from step S408 to step S402, the subjectluminance value Ex is obtained from the image signal recorded in stepS407. Therefore, in a case of using the image signal obtained in stepS404 from the first image sensor 103, the subject luminance value Ex iscaluculated by the first image signal processor 115 as described above.

In the above-described example, the subject luminance value Ex isobtained in step S402 and the imaging operation is switched based on thesubject luminance value Ex in step S403. However, switching may beperformed according to shooting conditions such as sensitivity set inthe image sensor. The imaging condition settings in that case may bedetermined by the controller 13 based on an image signal or aphotometric value calculated using an external measurement element suchas a photometric sensor (not shown), or may be determined by the user'soperation. Then, control may be done such that in a case where thesensitivity is larger than a predetermined sensitivity, the processproceeds to step S404, and in a case where the sensitivity is equal toor less than the sensitivity threshold, the process proceeds to stepS405.

Furthermore, the image sensor to be used may be switched according tothe aperture value set in accordance with the depth of field, shutterspeed, etc., the presence or absence of an ND filter for adjusting theamount of incident light, or the density of the inserted ND filter. Thatis, if the f-stop number narrower than a predetermined f-number is set,control is performed so as to use the first image sensor 103 (SPAD-typeimage sensor), and if the f-stop number wider than the predeterminedf-number is set, control is performed so as to use the second imagesensor 123 (CMOS type image sensor).

In addition, in a case where an ND filter is inserted in the light path,control is performed to use the first image sensor 103 (SPAD type imagesensor), and in a case where an ND filter is not inserted in the lightpath, control is performed to use the second image sensor 123 (CMOSimage sensor). Alternatively, in a case where an ND filter with adensity higher than a predetermined density is inserted in the lightpath, control is performed to use the first image sensor 103 (SPAD typeimage sensor), and in a case where an ND filter with a density lowerthan the predetermined density is inserted, control is performed to usethe second image sensor 123 (CMOS image sensor).

According to the fourth embodiment as described above, it is possible toprovide an image with good image quality by avoiding image deteriorationcaused by high luminance sinking due to count error in the SPAD typeimage sensor and deterioration in S/N ratio in the CMOS type imagingdevice.

Other Embodiment

The present invention may be applied to a system constituted by aplurality of devices or to an apparatus comprising a single device.

The present invention may be applied to a system constituted by aplurality of devices or to an apparatus comprising a single device.

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2018-149678, filed on Aug. 8, 2018 and No. 2019-119015, filed on Jun.26, 2019, which are hereby incorporated by reference herein in theirentirety.

What is claimed is:
 1. An image processing apparatus comprising at leastone processor programmed to perform operations of a generator thatgenerates an image having an expanded dynamic range using a first imagesignal and a second image signal, wherein the first image signal isoutput from a first image sensor having a plurality of pixels eachcounts a number of entering photons and outputs a count value, and thesecond image signal is output from a second image sensor having aplurality of pixels each outputs an electric signal corresponding to acharge amount obtained by performing photoelectric conversion onentering light.
 2. The image processing apparatus according to claim 1,wherein the generator uses the first image signal for a part of theimage where a signal level corresponding to the entering light is lessthan a predetermined first threshold value and uses the second magesignal for a part of the image where a signal level corresponding to theentering light is equal to or greater than a predetermined secondthreshold value.
 3. The image processing apparatus according to claim 2,further comprising at least one processor programmed to performoperations of an obtaining unit that obtains a luminance value as thesignal level corresponding to the entering light.
 4. The imageprocessing apparatus according to claim 3, wherein the generatorperforms processing of selecting the first image signal in a case wherethe luminance value is less than a predetermined first threshold value,selecting and synthesizing the first and second image signals in a casewhere the luminance value is equal to or greater than the predeterminedfirst threshold value and less than a predetermined second thresholdvalue which is larger than the predetermined first threshold value, andselecting the second image signal in a case where the luminance value isequal to or greater than the predetermined second threshold value. 5.The image processing apparatus according to claim 3, wherein theobtaining unit obtains the luminance value for each pixel.